Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial work towards a parallel dub building by supporting deferred logging #2717

Closed
wants to merge 1 commit into from
Closed

Initial work towards a parallel dub building by supporting deferred logging #2717

wants to merge 1 commit into from

Conversation

rikkimax
Copy link
Contributor

This introduces the deferred logging struct, unless enabled it should act identical to how it has in the past.

The idea is we would swap every function that ProjectGenerator.generate calls to use the DereferredLogState by ref instead of the global functions.

When parallelized we would turn on the deferment and once all the builds have completed for a given parallel set it would print them so that they will not have cross pollution between the builds.

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 20, 2023 •
edited
Loading

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 14

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5331080 bin/dub
-rough build time=69s
+executable size=5367944 bin/dub
+rough build time=66s
Full build output 
DUB version 1.34.0, built on Oct 15 2023
LDC - the LLVM D compiler (1.35.0):
  based on DMD v2.105.2 and LLVM 16.0.6
  built with LDC - the LLVM D compiler (1.35.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.35.0-rc.1+commit.6.ge8a57d4f: building configuration [application]
source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp`
source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp`
source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5367944 bin/dub
STAT:rough build time=66s

@rikkimax rikkimax force-pushed the prepare-parallel-logging branch from 850b38a to 9366773 Compare November 20, 2023 07:00
@rikkimax rikkimax changed the base branch from master to stable November 20, 2023 08:57
@PetarKirov
Copy link
Member

This design decision looks odd to me. I can't think of any build system (all of which build independent jobs in parallel) that defer logging until after a job is finished. Typically you'd have the output from different jobs running parallel prefixed with the name of the job and often also colorized differently to make the it easy to follow visually.

@PetarKirov
Copy link
Member

E.g. nix build with nix-output-monitor, which I use almost every day:
https://asciinema.org/a/7hJXH2iFLEkKxG1lL25lspqNn

@rikkimax
Copy link
Contributor Author

That sounds like it has good tradeoffs too.

I'm happy to see that implemented, although I feel I've done my part for the initial work on parallelization.

@rikkimax rikkimax closed this Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rikkimax @PetarKirov